Retropikzel's website - Blog - 2024-06-23 - Investigating how to implement callbacks on r7rs-pffi and some reasoning behid why the library exists

I dont usually make notes when I'm designing or investigating stuff, but I thought maybe it's time to start writing things down. After all it could be helpfull to others, or myself in the future when I have forgotten everything like I often do.

Premise

This posts topic is about how to implement callbacks for r7rs-pffi. The need for it arise when making an example of how to use it with libcurl. Up to that point I've mostly used the library with SDL2 or libc, and had not run into callbacks. I am also not a C developer so I have no knowledge about how often callbacks are used so I assumed the library could go without them.

Libcurl however is so important that before going forward with anything else building on top of the pffi library some solution must be at least considered.

I will also spread light about the design goals, requirements and hopes/wishes of the whole library.

Challenges and requirements

Implementations

The top priority for portable foreign function interface library is to work same way and have the same interface and behavior on every implementation that it supports. My personal opinion is also that the supported implementation list should be longer than 2-3 implementations. And span over multiple operating systems and "worlds".

If some implementations do not support callbacks then I have strong preference to not have callbacks at all. It's better to have something that supports a lot of implementations than everything that supports just couple implementations. Obviously where to exactly balance this is hard.

Operating systems

Guile and Sagittarius cover Linux nicely.

Support for Sagittarius and Racket are important because those implementations work on windows easily. I think guile also works on windows with cygwin, but these two have (IMO) easier installers. They also work on wine so I can test them without actually running windows myself.

Kawa, as it runs on JVM, works on multiple operating systems and I's also important in widening the usability of this library.

Platforms/ecosystems/worlds and future implementations support

By platforms/ecosystems/worlds I mean certain virtual machines or languages. Relevant to this post are

Some may disagree with my terminology or explanation here, but what I mean is that if you have experience in languages like C and C++ for example you are familiar with that "world". If you have experience with Java, Clojure, Kotlin or Groovy for example you are familiar with JVM "world". And if you have experience with backend Javascript for example you are familiar with the NodeJS "world".

Note that r7rs-pffi does not currently support any Scheme implementaton that runs on javascript, but I have my eyes on Biwasceme and LIPS. When either, or both, have support for (import (library name)) and cond-expand I will add support for them in this library. What this means in practice that there has to exists a nodejs FFI library that supports callbacks.

And you might ask: why are these "worlds" important? Where is this all headed? Why think so much about platforms, operating systems and compability and such?

Use cases and needs

Let me give you an example of use case that I had in mind when I started all this. Suppose you want to make a game with your favorite programming language Scheme. You want to use SDL2 which is a good choice.

Scenario A

You choose implementation A and library A-SDL2 which runs on OS-A. You develop your awesome game and show it to people. They want to play it, now, they run OS-B but your Scheme implementation does not support it. Sad times. :(

Despite the sadness you power trough, time comes to add physics to your game. But implementation A does not have any physics libraries. You think about porting your game to implementation B, but that implementation does not have SDL2 library.

I think this is the point many people possibly give up on Scheme, and I feel you. It can be painfull and frustrating not to be able to "just make a game". But lets say we had portable SDl2 library? How would the scenario look then?

Scenario B

You choose implementation A which runs on OS-A. You use SDL2 trough r7rs-pffi. You develop your awesome game and show it to people. They want to play it, now, they run OS-B but your Scheme implementation does not support it. You tell them to use implementation B which is supported by r7rs-pffi and runs on OS-B. Your game runs. Happy times. :)

Now the time comes to add physics to your game, you use r7rs-pffi for it and continue making the game.

Scenario C

You choose implementation C which runs on OS-A and OS-B but does not have SDl2 library. You use SDL2 trough r7rs-pffi. You develop your awesome game and show it to people. They want to play it. Your game runs. Happy times. :)

Now time comes to add physics to your game, but you dont want to use r7rs-pffi for it. Or more specically you dont need to use r7rs-pffi for it. The reason SDL2 needs to be used trough r7rs-pffi is that it interacts with the lower level stuff. If in this Scenario the implementation C is Kawa then you could use some Java library for physics.

Using Java library for it would obviously lock you into just using Kawa, but along the way before you had much more choices and still have the choice of not locking yourself into one implementation.

This last scenario is also important if you happen to have experience in the JVM world, you can use libraries that you are familiar with still benefit from the speed of the SDl2.

Current support for callbacks in Scheme implementations FFI's

Now bear in mind trough all this that my experience with C callbacks is around 0. :D

Currently supported implementations

Sagittarius

r7rs-pffi is hugely molded by Sagittarius FFI. The reason is that it runs on both Linux and Windows. And Sagittarius has this:

Macro make-c-callback return-type argument-types proc

There is also a comment on the example:

 ;; if you need pass a callback procedure, 'callback' mark needs to
 ;; be passed to the arguments list

so we might need a new pffi type called callback, on most implementations it propably just maps to pointer type.

Guile

Guile has this:

Scheme Procedure: procedure->pointer return-type proc arg-types
C Function: scm_procedure_to_pointer (return_type, proc, arg_types)

    Return a pointer to a C function of type return-type taking arguments of types arg-types
    (a list) and behaving as a proxy to procedure proc. Thus proc’s arity, supported argument
    types, and return type should match return-type and arg-types.

Kawa

The support for Kawa is implemented with JEP 442: Foreign Function & Memory API. My assumption is that it's possible. It would be quite weird if they did not think about it. This documentation seems to talk about it.

I did not investigate this very much, I'm quite confident that it's possible.

Racket

Racket has this:

(ffi-callback       proc
        in-types
        out-type
     [  abi
        atomic?
        async-apply
        varargs-after])     →       ffi-callback?

      proc : procedure?
      in-types : any/c
      out-type : any/c
      abi : (or/c #f 'default 'stdcall 'sysv) = #f
      atomic? : any/c = #f
      async-apply : (or/c #f ((-> any) . -> . any) box?) = #f
      varargs-after : (or/c #f positive-exact-integer?) = #f

The symmetric counterpart of ffi-call. It receives a Racket procedure and creates a
callback object, which can also be used as a C pointer.

Chicken

Chicken has foreign-safe-lambda. Which states:

To enable an external C function to call back to Scheme, the form foreign-safe-lambda
(or foreign-safe-lambda*) has to be used. This generates special code to save and restore
important state information during execution of C code. There are two ways of calling Scheme
procedures from C: the first is to invoke the runtime function C_callback with the closure
to be called and the number of arguments. The second is to define an externally visible
wrapper function around a Scheme procedure with the define-external form.

This highlight the importance of having Chicken supported early on. The specifics for implementations that compile to C are little bit different from those that have dynamic FFI and that shapes the design a lot.

From previous implementations one can start to imagine that maybe the solution could be something that would return function pointer on many implementations, like:

(pffi-procedure->callback (lambda (a b) (+ a b))

But then with chicken you would need:

(pffi-procedure->callback (foreign-safe-lambda (a b) (+ a b))

So maybe solution could look something like:

(pffi-callback-define plus (lambda (a b) (+ a b)))

but then assuming plus now is function pointer on many implementations calling it normally would propably fail, but on chicken it might work. Which would not be ideal.

Implementations with support in progress

STKlos

Now STKlos is the weakest link here, the documentation states that:

Note that the support for FFI is still minimal and that it will evolve in future versions.

Which is understanable. Many Scheme implementations can be expanded with for example hacking them and adding C code that way so need for dynamic FFI is often seen as not necessary. I'm glad tho that the FFI is still quite okay. Thought when Implementing support I have used undocumented functions to get around some issues.

I would love to have STKlos support as they have implemented many SRFI's and recently added support for R7RS. I have not yet used STKlos much but with addition of that I wished I could start.

Possible solutions for this is to patch STklos myself, would be nice way to learn more C, or ask about the maintainers plan regarding the FFI. Since it's "temporary" it's also possible that in the future STKlos will have FFI that also supports callbacks.

Cyclone

The Cyclone Foreign library does not seem have anything about callbacks. Reading the source leads me to believe that it's true.

Gambit

I always find researching Gambit stuff quite hard. The manual seems to be light. I've had a good experience with asking about stuff on their gitter thought. I searched that channel and came accross this (names removed because not relevant):

<name removed>:
Question re: the FFI and C function pointers. If I have this in C,
int cfcn (int (*cb)()) {
        cb();
}
and this in scheme,
(define call-cfcn
  (c-lambda ((function () int)) int "cfcn"))

(c-define (scmcallback) () int "cname" ""
  (display " ** scheme callback\n"))

(call-cfcn scmcallback)
I receive the message, 'ERROR (Argument 1) Can't convert to C function' upon executing
instead of the ' ** scheme callback' message. What have I missed or mis-read?

<name removed>:
I’m not sure which version of Gambit you are using, but this works fine:
$ gsc -v
v4.9.4-39-g9a887b80 20220503145411 x86_64-apple-darwin21.4.0 "./configure '--enable-single-host' 'CC=gcc-9'"
$ gsc cffi.scm
$ gsi cffi.o1
 ** scheme callback
 42
 $ cat cffi.scm
 (c-declare "

 int cfcn (int (*cb)()) {
       return cb() * 7;
 }

 ")

 (define call-cfcn
   (c-lambda ((function () int)) int "cfcn"))

 (c-define (scmcallback) () int "cname" ""
   (display " ** scheme callback\n")
     6)

 (pp (call-cfcn scmcallback))

from that I'm making an assumption that callbacks are possible.

Support waiting for the implementation

LIPS and Biwascheme

As we are calling C function r7rs-pffi on LIPS would only work on NodeJS. Searching npm with "ffi" gives at least these libraries as result:

aaand many others, I'm confident that there will be at least one library that fits our use case.

What would the interface look like?

Ideally something like this:

(define plus (lambda (a b) (+ a b)))
(pffi-define uses-callback clib 'uses-callback 'int (list 'pointer))
(uses-callback (procedure->pointer plus))

On implementations which have something like (procedure->pointer...) this would propably work. I think it would also be good have a scheme procedure which can be used everywhere and then when passing it we would just take the pointer.

On Chicken scheme however this does not work, as it would need to be like this:

(foreign-safe-lambda plus (lambda (a b) (+ a b)))
(pffi-define uses-callback clib 'uses-callback 'int (list 'pointer))
(uses-callback pointer plus)

And now that I think about it could make a mess too, if C passes pointers to the callback function then if you want to use it like a normal C function you would need to convert the arguments to pointers anyway.

So maybe it is best have something like this:

(pffi-define-callback plus (lambda (a b) (+ a b)))
(pffi-define uses-callback clib 'uses-callback 'int (list 'pointer))
(uses-callback plus)

and then tell the users not to use it as anything else than a callback.

Conclusion

There was much wider support that I thought, which is excellent news. I anticipated that there would be much more tougher choices but for now it only seems that STKlos and Cyclone support needs to be dropped. Their support was not ready yet, so not much is lost. And in the future support can hopefully be added.

Now all thats left is to implement the support and then do a new updated post about how to use libcurl.